Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework.

OBJECTIVE: Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare. DESIGN: We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made. SETTING: Online using ChatGPT web interface. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework. RESULTS: ChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT's accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=-0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors. CONCLUSION: ChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT's errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

Monitoring real-time junior doctor sentiment from comments on a public social media platform: a retrospective observational study.

OBJECTIVES: To investigate whether sentiment analysis and topic modelling can be used to monitor the sentiment and opinions of junior doctors. DESIGN: Retrospective observational study based on comments on a social media website. SETTING: Every publicly available comment in r/JuniorDoctorsUK on Reddit from 1 January 2018 to 31 December 2021. PARTICIPANTS: 7707 Reddit users who commented in the r/JuniorDoctorsUK subreddit. MAIN OUTCOME MEASURE: Sentiment (scored -1 to +1) of comments compared with results of surveys conducted by the General Medical Council. RESULTS: Average comment sentiment was positive but varied significantly during the study period. Fourteen topics of discussion were identified, each associated with a different pattern of sentiment. The topic with the highest proportion of negative comments was the role of a doctor (38%), and the topic with the most positive sentiment was hospital reviews (72%). CONCLUSION: Some topics discussed in social media are comparable to those queried in traditional questionnaires, whereas other topics are distinctive and offer insight into what themes junior doctors care about. Events during the coronavirus pandemic may explain the sentiment trends in the junior doctor community. Natural language processing shows significant potential in generating insights into junior doctors' opinions and sentiment.

Ultrasound-guided motor unit scanning electromyography.

INTRODUCTION/AIMS: Measuring the spatial dimensions of a single motor unit remains a challenging problem, and current techniques, such as scanning electromyography (EMG), tend to underestimate the true dimensions. In this study we aimed to estimate more accurately the dimensions of a single motor unit by developing a clinically applicable scanning EMG protocol that utilizes ultrasound imaging to visualize and target a transect through the center of a single motor unit. METHODS: Single motor unit twitches in the tibialis anterior muscles of healthy volunteers were elicited via stimulation of the fibular nerve, visualized with ultrasound, and targeted with an intramuscular EMG electrode. The electrode was moved by hand in small steps through the motor unit territory. Ultrasound video output was synchronized to EMG capture, and the needle position was tracked at each step. RESULTS: Eight recordings from six participants were collected. The technique was quick and easy to perform (mean time, 6.1 minutes) with reasonable spatial resolution (mean step size, 1.85 mm), yielding motor unit territory sizes between 1.53 and 14.65 mm (mean, 7.15 mm). DISCUSSION: Ultrasound-guided motor unit scanning EMG is a quick and accurate method for obtaining a targeted motor unit transect. This combination of two readily available clinical tools provides insights into the dimensions and internal structure of the motor unit as a marker for neuromuscular conditions.

Serotonin syndrome.

The serotonin syndrome is a life-threatening adverse drug reaction resulting from excess serotonergic agonism due to interactions between multiple drugs, poisoning, or less commonly due to therapeutic action of a single drug. The central triad of features in serotonin syndrome are altered mental state, autonomic hyperactivity, and neuromuscular abnormalities in the context of a patient with new/altered serotonergic therapy, although not all these features are consistently present in all patients. The severity of serotonin syndrome can be assessed clinically based on the number and severity of features. Severe serotonin syndrome warrants more careful management on a high-dependency unit. In case of temperature exceeding 38.5°C, urgent cooling measures and sedation should be employed, progressing to rapid sequence intubation and paralysis if cooling measures are ineffective.

Electrical cross-sectional imaging of human motor units in vivo.

OBJECTIVES: In many neuromuscular diseases, weakness results from a disruption in muscle fibres' arrangement within a motor unit. Limitations in current techniques mean that the spatial distribution of fibres in human motor units remains unknown. METHODS: A flexible multi-channel electrode was developed and bonded to a clinical electromyography (EMG) needle. Muscle fibre action potentials were localised using a novel deconvolution method. This was tested using simulated data, and in recordings collected from the tibialis anterior muscle of healthy subjects. RESULTS: Simulated data indicated good localisation reliability across all sections of the electrode except the end sections. A corrected fibre density was estimated up to 1.4 fibres/mm2. Across five recordings from three individuals, between 4 and 14 motor units were detected. Between 1 and 20 muscle fibres were localised per motor unit within the electrode detection area, with up to 220 muscle fibres localised per recording, with overlapping motor unit territories. CONCLUSIONS: We provide the first direct evidence that human motor units spatially overlap, as well as data related to the spatial arrangement of muscle fibres within a motor unit. SIGNIFICANCE: As well as providing insights into normal human motor physiology, this technology could lead to faster and more accurate diagnosis in patients with neuromuscular diseases.

Assessing inter-rater reproducibility in MScanFit MUNE in a 6-subject, 12-rater "Round Robin" setup.

OBJECTIVE: To assess the inter-rater reliability of MScanFit MUNE using a "Round Robin" research design. METHODS: Twelve raters from different centres examined six healthy study participants over two days. Median, ulnar and common peroneal nerves were stimulated, and compound muscle action potential (CMAP)-scans were recorded from abductor pollicis brevis (APB), abductor digiti minimi (ADM) and anterior tibial (TA) muscles respectively. From this we calculated the Motor Unit Number Estimation (MUNE) and "A50", a motor unit size parameter. As statistical analysis we used the measures Limits of Agreement (LOA) and Coefficient of Variation (COV). Study participants scored their perception of pain from the examinations on a rating scale from 0 (no pain) to 10 (unbearable pain). RESULTS: Before this study, 41.6% of the raters had performed MScanFit less than five times. The mean MUNE-values were: 99.6 (APB), 131.4 (ADM) and 126.2 (TA), with LOA: 19.5 (APB), 29.8 (ADM) and 20.7 (TA), and COV: 13.4 (APB), 6.3 (ADM) and 5.6 (TA). MUNE-values correlated to CMAP max amplitudes (R2-values were: 0.463 (APB) (p<0.001), 0.421 (ADM) (p<0.001) and 0.645 (TA) (p<0.001)). The average perception of pain was 4. DISCUSSION: MScanFit indicates a high level of inter-rater reliability, even with only limited rater experience and is overall reasonably well tolerated by patients. These results may indicate MScanFit as a reliable MUNE method with potential as a biomarker in drug trials.

Ipsilateral Motor Evoked Potentials as a Measure of the Reticulospinal Tract in Age-Related Strength Changes.

Background: The reticulospinal tract (RST) is essential for balance, posture, and strength, all functions which falter with age. We hypothesized that age-related strength reductions might relate to differential changes in corticospinal and reticulospinal connectivity. Methods: We divided 83 participants (age 20-84) into age groups <50 (n = 29) and ≥50 (n = 54) years; five of which had probable sarcopenia. Transcranial Magnetic Stimulation (TMS) was applied to the left cortex, inducing motor evoked potentials (MEPs) in the biceps muscles bilaterally. Contralateral (right, cMEPs) and ipsilateral (left, iMEPs) MEPs are carried by mainly corticospinal and reticulospinal pathways respectively; the iMEP/cMEP amplitude ratio (ICAR) therefore measured the relative importance of the two descending tracts. Grip strength was measured with a dynamometer and normalized for age and sex. Results: We found valid iMEPs in 74 individuals (n = 44 aged ≥50, n = 29 < 50). Younger adults had a significant negative correlation between normalized grip strength and ICAR (r = -0.37, p = 0.045); surprisingly, in older adults, the correlation was also significant, but positive (r = 0.43, p = 0.0037). Discussion: Older individuals who maintain or strengthen their RST are stronger than their peers. We speculate that reduced RST connectivity could predict those at risk of age-related muscle weakness; interventions that reinforce the RST could be a candidate for treatment or prevention of sarcopenia.

Sarcopenia and frailty in individuals with dementia: A systematic review.

BACKGROUND: Population aging has resulted in an increase in age-related conditions. Sarcopenia, the progressive loss of muscle mass and strength, and frailty, vulnerability to poor resolution of homeostasis after a stressor, are common causes of functional decline in older individuals. There is a paucity of work on how they interrelate with dementia. The objective of this review was to examine the literature on sarcopenia and frailty in dementia. METHODS AND RESULTS: Studies of sarcopenia and frailty in dementia were searched for in EMBASE, PubMed and Web of Science, and via hand-searching. Citations were screened for independently by two reviewers, with disagreements resolved by a third reviewer. To be eligible for inclusion, the articles needed to fulfil: (1) English language; (2) human studies; and (3) full-text available. Dementia of any aetiology was included. 303 non-duplicate recorders were identified, of which 270 irrelevant papers were excluded. Of the remaining 33, 27 examined frailty and 13 sarcopenia, with six of these measuring both. An increased prevalence of frailty and sarcopenia was noted in dementia patients. However, nine papers did not specify dementia aetiologies. Of those that did (n = 21), 20 examined Alzheimer's disease, with three also including Lewy body dementia, three vascular dementia, and one Parkinson's disease dementia. CONCLUSION: Most papers examined frailty, rather than sarcopenia, in dementia. The studies were heterogeneous, using different protocols and non-validated definitions. However, dementia patients may have an increased prevalence of frailty and sarcopenia. This review highlights key gaps in accurate diagnosis of frailty/sarcopenia and in non-Alzheimer's dementia.